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- ■ This  paper  discusses  the  status  of  performance  measurement  (PM)  for  maintenance.  During  and  after  World 
War  II,  both  Navy  and  Air  Force  maintenance  training  programs  made  extensive  use  of  formal  job  task  performance 
tests.  But  for  economy  reasons,  these  tests  were  later  abandoned  in  favor  of  paper-and-pencil  theory  and  job 
knowledge  tests.  Considering  the  results  of  later  research,  these  actions  were  most  unfortunate.  This  research  has 
indicated  that  such  paper-and-pencil  tests  do  not  indicate  how  well  individuals  can  perform  the  tasks  of  their  jobs. 
Even  though  FM  iitere  used  extensively  during  and  after  World  War  II,  there  have  bwn  few  systematic  research  and 
development  U RADI  efforts  concerning  the  refinerrrent  of  PM  for  maintenance.  This  paper  briefly  describes  the 
APHRL^RA^fTorts  for  PM  which  have  given  due  consideration  to  the  man-machine  interface.  The  rather  promising 
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SUMMARY 


My  paper  discusses  the  status  of  performance 
measurement  (PM)  for  maintenance. 

1.  During  and  after  World  War  II,  both  Navy 
and  Air  Force  maintenance  training  programs 
made  extensive  use  of  formal  job  task  performance 
tests  (JTPT).  But  for  economy  reasons,  these  tests 
were  later  abandoned  in  favor  of  paper-and-pencil 
theory  and  job  knowledge  tests. 

2.  Considering  the  results  of  later  research, 
these  actions  were  most  unfortunate.  This  research 
has  indicated  that  such  paper-and-pencil  tests  do 
not  indicate  how  well  individuals  can  perform  the 
tasks  of  their  jobs.  Table  1 of  my  paper  indicates 
the  correlations  obtained  from  several  studies 
which  have  compared  JTPT  to  theory  tests  and  to 
job  knowledge  tests.  The  table  also  includes 
correlations  of  JTPT  with  school  marks.  None  of 
these  substitute  measures  are  sufficiently  valid  for 
use  as  substitutes  for  JTPT.  1 am  convinced  that 
the  current  unquestioned  usage  of  such 
paper-and-pencil  tests  in  Held  and  training 
situations  would  be  unforgivable  if  the  people 
involved  understood  that  their  current  testing 
practices  are  really  invalid.  No  matter  how  cheaply 
paper-and-pencil  job  knowledge  tests  can  be 
prepared  or  how  easily  they  can  be  administered, 
such  tests  are  not  a bargain.  Their  results  are 
almost  meaningless  in  temis  of  ability  to  perform 
maintenance  tasks. 

3.  Even  though  PM  was  used  extensively 
during  and  after  World  War  II  and  even  thou^ 
many  valuable  ad  hoc  PM  efforts  have  been 
reported  by  the  Army,  Navy,  and  Air  Force,  there 
have  been  few  systematic  research  and 
development  ( R&.D)  efforts  concerning  the 
refinement  of  PM  for  maintenance.  A notable 
exception  was  the  work  of  the  Air  Force  Personnel 
and  Training  Research  Center  (AFPTRC) 
Maintenance  Laboratory  at  Lowry  AFB  in  the 
middle  50’s.  Another  more  recent  systematic 
Army  effort  was  the  Vineberg  effort, 
accomplished  by  HumRRO  at  Presidio  of 
Monterey,  California  in  the  early  70’s. 

4.  My  papier  briefly  describes  the  AFHRL 
efforts  which  have  given  consideration  to  the 
man-machine  interface.  One  important  result  of 
this  consideration  has  been  the  evolution  and 
articulation  of  a structure  for  handling 
maintenance  functions  and  their  complex 
relationships  in  a systematic  manner.  This 
structure  includes  (a)  standard  maintenance 
functions  and  action  verbs,  (b)  a working 


definition  of  a maintenance  task,  and  (c)  schemes 
for  handling  the  complexities  of  maintenance 
tasks.  In  the  interest  of  time,  1 will  limit  my 
comments,  here,  to  the  task  definition  and  a 
scheme  for  indicating  dependencies  among  task 
functions. 

5.  Within  the  list  of  action  verbs  are  a number 
of  key  action  verbs  (functions);  such  as  checkout, 
align,  adjust,  calibrate,  remove,  replace,  and 
troubleshoot.  A key  action  verb,  with  an 
appropriate  spiecific  hardware  unit  as  its  predicate, 
becomes  a task  statement.  Such  a task  statement 
represents  a maintenance  task  which  can  be 
demanded  by  the  existence  and  operapipn  of  a 
specific  machine  subsystem.  A list  of  these 
functions  is  found  in  AFHRL-TR-7343(I)  (Joyce 
et  al.,  1973).  This  fist  includes  functions  which  are 
found  in  both  mechanical  and  electronic  jobs. 
Some  apply  to  only  mechanical  jobs  and  some 
apply  to  both. 

6.  Another  matter  of  serious  concern  when 
developing  and  structuring  PM  for  maintenance 
tasks  is  the  interaction  among  the  maintenance 
tasks  for  identical  hardware.  The  scheme  reflecting 
a four-level  hierarchy  of  depiendencies  was 
developed,  figure  5 gives  a grapliic  presentation  of 
these  dependencies  among  maintenance  activities 
for  an  electronic  hardware.  An  exam  pile  of  the 
depiendency  relationship  is  troubleshooting,  which 
may  include  all  of  the  activities  above  it. 

7.  Starting  in  1969,  the  Advanced  Systems 
Division  of  the  AFHRL  supported  a modest 
program  to  provide  the  Air  Force  with  the 
necessary  tools  for  measuring  the  ability  of 
maintenance  personnel  to  perform  the  key  tasks  of 
their  jobs.  The  contractor  for  this  work  was  Matrix 
Research  Company.  The  scope  of  this  work  was 
limited  to  the  maintenance  of  electronic  hardware 
at  the  organizational  and  intermediate  levels.  This 
program  had  two  objectives:  (a)  to  develop  a 
model  battery  of  JTPT  together  with  appropriate 
scoring  schemes  for  the  measurement  of  the  task 
performance  ability  of  electronic  maintenance 
personnel  (an  effort  was  to  be  made  for  the 
development  of  JTPT  which  could  be  easily 
administered),  and  (b)  using  the  JTPT  of  this 
battery  as  criteria,  to  develop  and  try  out  a series 
of  pap>er  and  piencil  symbolic  substitute  tests 
which  would  hopiefully  have  high  empirical 
validity. 

8.  A model  battery  of  48  criterion  referenced 
JTPT,  and  a test  administrator’s  handbook  were 
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developed  for  nreasuring  ability  to  perform 
electronic  maintenance  tasks.  Copies  of  the  actual 
instructions  for  test  subjects  together  with  the  test 
administrator's  handbook  are  available  from  the 
Defense  Documentation  Center  (DDC).  The  test 
administrator's  handbook  was  developed  with 
step-by-step  detailed  instructions  so  that  an 
individual  with  a minimum  of  electronic 
maintenance  experience  could  administer  the  tests. 

9.  After  considering  product,  process,  and 
time  as  to  their  appropriateness  for  scoring  the 
results  for  each  activity,  it  was  decided  that  a test 
subject  had  not  reached  criterion  until  he  had 
produced  a complete,  satisfactory  product.  This 
was  a go,  no-go  criterion.  Table  2 summarizes  the 
number  of  tests,  problems  and  scorable  products 
by  class  developed  for  the  Doppler  Radar 
AN/APN-147  and  Computer  AN/ASN-35.  The 
simple  addition  of  numbers  shown  indicates  that 
there  are  48  tests,  8 1 problems,  and  1 33  scorable 
products.  But,  these  numbers  tell  us  nothing  in 
terms  of  the  content  of  the  tests.  To  say  that  one 
test  subject  accomplished  100  scorable  products 
while  another  accomplished  90,  tells  us  nothing 
about  the  job  readiness  of  these  individuals  or 
about  whether  one  is  better  than  the  other.  The 
varieties  of  scorable  products  are  so  diverse  that 
any  combination  of  them,  without  regard  to  what 
they  represent,  is  meaningless. 

10.  The  only  nieaningful  presentation  of  such 
information  must  be  in  terms  of  a profile  designed 
to  attach  meaning  to  such  numbers.  A sample  of 
part  of  such  a profile  is  shown  on  Figure  6.  This 
profile  is  not  presented  as  the  final  solution  to  the 
profile  problem  for  JTPT  for  electronic 
maintenance.  It  does  contain  most  of  the 
important  information  regarding  a test  subject's 
success  on  the  full  range  of  tests.  It  gives  a 
meaningful  picture  of  the  subject's  job  task 
abilities  as  measured  by  the  test  battery,  indicating 
the  subject's  strengths  and  weaknesses.  The  subject 
receives  no  “credit”  for  a problem  unless  he 
obtains  all  of  the  expected  products.  No  attempt  is 
made  to  combine  these  scores  in  terms  of 
meaningless  numbers. 

11.  There  is  no  doubt  that  a battery  of  JTPT 
would  require  more  training  and  on-the-job  time 
of  the  test  subjects,  more  equipment,  and  specially 
trained  test  administrators.  It  will  be  recalled  that 
these  were  high  among  the  reasons  given  for 
dropping  PM  from  the  Air  Force  and  Navy 
maintenance  training  'programs.  Therefore,  the 
availability  of  empirically  valid  symbolic  substitute 
tests  would  be  highly  desirable.  Even  though 
previous  attempts  to  develop  such  tests  as  the  Tab 
test  (Crowder,  Morrison,  & Demaree,  1954)  had 


failed,  it  was  our  opinion  that  much  more  work 
could  be  done  to  improve  symbolic  maintenance 
tests  as  substitutes  for  JTPT.  It  was  hypothesized 
that  higher  correlations  could  possibly  be  obtained 
by  a different  approach  to  the  development  of 
symbolic  tests.  For  example,  we  felt  that  higher 
correlations  could  be  obtained  by  adding  realistic 
clutter  to  the  cognitive  aspects  of  troubleshooting 
tests,  such  as,  using  test  equipment  to  obtain  test 
point  information.  A companion  graphic  symbolic 
test  was  developed  for  each  of  the  job  activities  for 
which  a criterion  referenced  JTPT  had  previously 
been  developed.  Based  on  two  limited  validations, 
all  of  the  graphic  symbolic  tests,  with  the 
exception  of  the  symbolic  test  for  soldering, 
indicated  sufficient  promise  to  justify  further 
consideration  and  rePinement.  Due  to  a shortage  of 
available  subjects,  the  number  of  pairs  of  subjects 
was  extremely  small.  All  of  these  promising 
graphic  symbolic  tests,  therefore,  must  be  given 
more  extensive  validations  using  larger  numbers  of 
experienced  subjects.  An  attempt,  also,  was  made 
to  develop  video  symbolic  substitute  tests,  but  this 
effort  produced  no  promising  results. 

12.  My  paper  also  discusses  several  problems 
concerning  the  research,  development,  and 
implementation  of  PM.  There  is  no  doubt  that 
there  is  a great  need  for  PM  in  maintenance.  One 
of  the  greatest  problems  is  to  develop  the  demand 
pull  or  necessary  want  to  get  newly  developed 
technologies  such  as  PM  institutionalized.  This  is 
especially  difficult  when  a technology  requires 
fundamental  changes  in  long  existing  programs, 
procedures,  and  attitudes  of  entrenched 
establishments.  Operational  organizations 
invariably  attempt  to  implement  a much  “watered 
down”  version  of  the  technology  and, 
consequently,  obtain  greatly  “watered  down” 
results.  In  some  cases,  only  cosmetic  changes  to 
existing  programs  are  reported  as  implementations. 
Currently  it  requires  years  of  persistent  effort  (or 
push)  on  the  part  of  the  research  community  to 
get  a technology  properly  institutionalized.  A 
mechanism  must  be  developed  for  the  timely 
implementation  of  each  new  technology  to  ensure 
its  integrity.  A mechanism  similar  to  that  used  for 
new  weapons  systems  is  recommended.  Such  a 
mechanism  must  make  efficient  and  effective  use 
of  the  “know-how”  of  the  developers  of  the 
technology  and  make  them  responsible  and 
accountable  for  its  implementation.  A new 
technology  should  not  be  turned  over  to  a using 
command  for  its  operation  until  it  is  in  place,  “ 
debugged,”  and  operational-just  as  a new 
weapons  system  is  not  turned  over  to  an 
operational  command  until  it  has  been 
“debugged”  and  proven  to  be  ready  for 
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operational  use. 

13.  A number  of  related  problems  are  also 
discussed.  There  is  a well-developed 
paper-and-pencil  test  technology  which  is  based  on 
testing  theory  which  is  appropriate  for  the 
academic  variety  of  education.  This  technology 
has  been  institutionalized  and  is  well  entrenched  in 
the  DOD  personnel  and  training  systems.  All 
education  test  and  measurement  textbooks  and 
courses  reflect  this  technology.  Psychological 
measurement  texts  also  emphasize  this  technology. 
At  least  two  generations  of  teachers  and  test  and 
measurement  psychologists  have  been  trained  in 
the  use  of  this  technology  and,  as  a result,  many 
have  unquestioned  faith  in  its  application  to  any 
personnel  measurement  problem.  Most  of  these 
people  are  products  of  the  academic  world.  Few 
have  had  any  “hands-on”  experience  in  performing 
maintenance  tasks.  When  the  appropriateness  of 
their  technology  for  the  measurement  of 
maintenance  ability  is  questioned,  many  members 
of  this  paper-and-pencil  testing  establishnwnt 
become  threatened  and,  therefore,  defensive. 

14.  In  spite  of  the  extensive  military  history  of 
usage,  there  is  no  PM  establishment  comparable  to 
the  paper  and  pencil  test  establishment.  There  are 
no  college  test  and  measurement  courses  (even  in 
vocational  education  departments)  which  teach 
PM  technology,  and  there  are  no  textbooks 
devoted  to  the  subject.  The  vocational  educators 
have  emulated  their  academic  brethren  by  using 
their  measurement  texts.  And  there  has  only  been 
a limited  amount  of  systematic  R&D  concerning 
the  development  of  a PM  technology.  Most  of  the 
current  PM  technology  for  maintenance  is  found 
in  DOD  technical  reports. 

15.  Success  in  aircraft  pilot  training  and  other 
operator  training  has  always  been  based  on  PM; 
that  is,  demonstrated  ability  to  perform  key  job 
tasks.  Consequently,  these  training  programs  have 
been  designed  to  ensure  success  on  PM.  Such 
training  has  been  characterized  by  an  abundance 
of  supervised  practice  of  job  tasks.  But  for 
maintenance  personnel,  paper-and-pencil  theory 
and  job  knowledge  tests  have  been  used  as  the 
principal  means  for  determining  both  the  school 
and  job  success.  As  a result,  maintenance  training 
programs,  both  formal  courses  and  career 
development  courses  (CDC),  have  come  to  be 
structured  to  ensure  success  on  paper-and-pencil 
tests.  This  has  resulted  in  the  greater  part  of  many 
so-called  maintenance  training  courses  taking  on 
the  verbal  characteristics  of  academic  education. 
And  this  has  happened  at  the  expense  of 
supervised  practice  of  job  tasks. 


16.  The  administration  of  PM  requires  time. 
Timewise,  it  certainly  would  be  impossible  to 
administer  a PM  to  a maintenance  man  for  every 
possible  task  that  his  hardware  system  might 
produce.  This  world  of  tasks  and  people  must  be 
sampled.  The  model  PM,  described  previously, 
provides  a sampling  procedure  based  on  major  task 
functions  such  as  checkout,  align,  adjust, 
troubleshoot,  etc.  But  even  this  sampling  across 
possible  tasks  resulted  in  48  tests  and  1 33  scorable 
products.  It  would  be  impractical  to  give  any  one 
test  subject  all  of  these  48  tests  at  any  one  time. 
Systematic  sampling  schemes  must  be  developed 
across  tests.  The  purposes  for  which  PM  results  are 
to  be  used  should  be  considered  when  developing 
sampling  schemes.  Such  purposes  of  PM  could 
include  ascertaining  (a)  the  job  task  proficiency  of 
an  individual,  (b)  the  job  effectiveness  of  a training 
program,  and  (c)  the  proficiency  of  a maintenance 
unit.  Each  of  these  purposes  would  require  a 
different  mix  (or  mixes)  of  tests  and  people.  Some 
suggestions  for  such  samplings  can  be  found  in 
AFHRL-TR-74-57(ll)  Part  I (Shriver  & Foley, 
1974a).  But  it  sliould  be  remembered  that  these 
are  suggestions  that  must  still  be  field  tested,  in 
the  case  of  detemiining  unit  proficiency,  some  PM 
can  be  administered  by  on-line  observation  of 
tasks  that  are  often  repeated;  such  as,  checkout. 
However,  there  will  always  be  a requirement  for 
off-line  PM  concerning  critical,  but  seldom 
performed  tasks.  Whether  the  PM  is  performed 
on-line  or  off-line,  the  test  administrator  must  use 
the  same  objective  scoring  procedures,  the  criteria 
of  success  being  an  acceptable  product. 

17.  In  spite  of  all  of  the  evidence  supporting 
requirements  of  PM  for  maintenance,  it  has  been 
extremely  difficult  to  obtain  R&D  funding  for 
efforts  to  advance  the  PM  technology.  In  addition, 
difficulty  has  been  experienced  in  finding  and 
retaining  Air  Force  professionals  with  the 
necessary  capability  and  interest  to  pursue  an 
effective  PM  R&D  program  for  maintenance.  Such 
professionals  are  necessary,  either' for  an  in-house 
or  contractor  program.  Any  successful  program  in 
this  PM  area  must  be  a long  range  program  making 
use  of  existing  expertise  and  aimed  at  expanding 
such  expertise.  “Off  again,  on  again”  efforts 
and/or  jumping  to  a new  contractor  with  every 
start  will  result  in  little  improvement  in  PM 
technology. 

1 8 . Excessive  maintenance  costs  are  never 
going  to  be  reduced  as  long  as  we  don’t  have  JTPT 
and/or  empirically  valid  symbolic  substitutes  to 
ascertain  how  efficiently  maintenance  men 
perform  the  tasks  of  their  jobs.  In  my  opinion,  the 
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lack  of  such  measures  of  maintenance  performance 
is  a most  serious  deficiency  in  DOD.  As  such,  R&D 
in  this  area  should  have  an  extremely  high  priority, 
j For  a long  range  R&D  effort,  five  general  areas  of 

J concentration  ate  recommended;  namely: 

1.  Refinement  of  Model  JTPT  Battery 

I (Electronic  Maintenance). 

2.  Refinement  of  Symbolic  Substitutes 

; (Electronic  Maintenance). 

I 3.  Development  of  Model  JTPT  Battery 

(Mechanical  Maintenance). 

i 4.  Development  of  Symbolic  Substitutes 

i (Mechanical  Maintenance). 

[.  5.  Job  Aptitude  Test  Research  Based  on 

? Results  on  JT^. 

t 


19.  Probably  the  most  cost-effective  approach 
for  PM  R&D  (for  electronic  and  mechanical 
maintenance)  would  be  to  concentrate  on  the 
development  and  refinement  of  JTPT  on  the  use 
of  key  test  equipments  prior  to  proceeding  with 
the  other  task  functions  of  the  proposed  model 
test  batteries.  The  use  of  general  test  equipment  is 
a prerequisite  to  maintenance  task  functions  such 
as  alignment,  calibration  and  troubleshooting.  In 
addition,  general  test  equipments  usually  have 
wide  usage  in  such  task  functions  across  many 
hardware  systems  and  there  are  a substantial 
amount  of  data  which  indicate  that  many 
maintenance  men  are  weak  in  their  test  equipment 
ability.  So,  a general  improvement  in  ability  to  use 
test  equipment  is  an  important  and  necessary 
factor  for  the  general  improvement  of  several 
maintenance  task  functions. 
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PERFORMANCE  MEASUREMENT  OF  MAINTENANCE 


I.  PKRKORMANCE  MPASCREMENT  HISTORY 

Performance  Measurement  (PM)  is  not  some- 
thing new  for  the  Defense  Establishment.  But 
many  of  the  past  PM  efforts  were  not  adequately 
documented.  As  a result,  even  the  e.xistence  of 
many  efforts  has  been  forgotten.  From  personal 
experience  during  World  War  II,  I know  tliat  the 
training  establishments  of  both  the  Army  Air 
Force  and  the  Navy  made  extensive  use  of  such 
measurements  for  such  maintenance  job  tasks  as 
checkout,  alignment,  and  troubleshooting. 

1 do  not  know  exactly  when  PMs  were  dc- 
emphasized  in  Navy  maintenance  training 
programs.  However,  in  1962,  Harris  and  Mackie 
reported  why  PM  was  not  being  used  in  Navy 
training  and  field  activities.  Their  report  indicates 
that  PMs  were  generally  felt  to  require  too  much 
equipment  and  personnel  time  to  be  feasible. 

In  die  Air  Force,  an  active  and  substantial  PM 
program  continued  until  1956.  These  measure- 
ment programs  for  the  Air  Training  Command 
(ATC)  of  the  Air  Force  and  its  predecessor,  the 
Amiy  Air  Forces,  included  elaborate  checkrooms. 
To  increase  measurement  objectivity  and  decrease 
instructor  bias,  diese  checkrooms  were  manned  by 
full-time  test  administrators.  Tlieir  sole  job  was  to 
develop  and  administer  both  written  and  perform- 
ance tests.  In  most  cases,  these  checkrooms  were 
assigned  their  own  hardware  systems  or  sub- 
systems which  were  used  exclusively  for  PM.  The 
PM  required  a substantial  amount  of  equipment 
time,  as  well  as  test  subject  and  test  administrator 
time. 

In  1956,  almost  all  checkrooms  were  abolished 
to  save  money,  equipment,  and  personnel.  An 
often-used  argument  in  favor  of  this  action  was 
that  most  civilian  schools  did  not  have  checkrooms 
and  that  in  civilian  schools,  the  classroom,  labora- 
tory or  shop  instructors  were  responsible  for 
measurement  and  grading,  which  was  true.  But  the 
weakness  in  this  argument  is  that  in  most  cases, 
the  shop  instructors  in  civilian  vocational  schools 
did  not  have  time  to  administer  PM  and  also  super- 
vise shop  exercises.  As  a result,  the  Air  Force  had  a 
far  superior  and  more  valid  measuicment  system 
than  civilian  vocational  schools.  Another  argument 
was  that  the  resources  required  for  PM  could  not 
be  justified  since  PMs  were  not  part  of  the  directed 
mission  of  ATC. 

But  no  matter  what  the  reason,  there  was  a 
drastic  decrease  in  the  number  of  PMs  used  in  ATC 


after  1956.  Die  decrease  or  elimination  of  PMs 
resulted  in  complete  reliance  on  paper-and-pencil 
theory  and  job  knowledge  tests  as  measures  of 
school  success.  The  absence  of  PM  resulted  in  a 
decreased  emphasis  concerning  “hands-on”  equip- 
ment exercises  in  maintenance  training  programs. 
Tliis  was  especially  true  for  electronic  maintenance 
training. 

Early  Air  Force  R&D  for  Maintenance  PM 

Althougli  the  use  of  PM  in  ATC  did  encourage 
the  use  of  valuable  “hands-on”  training,  the  PM 
used  did  not  reflect  a systematic  development 
process.  As  a result,  their  quality  varied  greatly 
from  checkroom  to  checkroom.  These  and  other 
weaknesses  of  the  PM  used  in  ATC  were  recog- 
nized by  personnel  of  the  Maintenance  Laboratory 
of  the  Air  Force  Personnel  and  Training  Research 
Center  (AFPTRC)  in  the  early  1950’s.  (This 
Maintenance  Laboratory,  located  at  Lowry  Air 
Force  Base,  Colorado,  was  directed  by  Dr.  Robert 
M.  Gagne).  Tliis  measurement  research  and 
development  (R&D)  continued  until  the  demise  of 
that  laboratory  in  1958. 

One  output  of  this  effort  intended  for  improve- 
ment of  the  development  and  administration  of 
PM  was  “A  Guide  for  Use  in  Performance  Testing 
in  Air  Force  Technical  Schools”  (Highland,  1955). 
However,  this  useful  document  was  published  too 
late.  Due  to  the  closing  of  checkrooms  and  the 
resulting  deemphasis  of  PM,  this  guide  received 
little  or  no  use  in  ATC.  However,  if  it  had  been 
followed,  it  certainly  would  have  resulted  in 
improved  PM.  One  serious  shortcoming  of  this 
guide,  as  viewed  from  today’s  vantage  point,  was 
the  undue  credence  it  gave  paper-and-pencil 
measures. 

In  this  regard,  another  important  document  of 
the  Maintenance  Laboratory  reported  the  inter- 
correlations of  measures  coneeming  the  profi- 
ciency of  radar  mechanics  (Crowder  et  al.,  1954). 
Tliis  was  one  of  the  early  studies  which  reported 
extremely  low  correlations  between  results  of  PM 
and  results  of  paper-and-pencil  theory  and  job 
knowledge  tests.  During  the  19S0’s  and  early 
1960’s,  there  were  a number  of  other  studies  that 
produced  similar  findings.  This  matter  will  be 
discussed  later. 

It  certainly  was  unfortunate  for  the  quality  of 
maintenance,  that  the  use  of  PM  was  deem- 
phasized.  But  at  the  time  of  these  actions,  much  of 
the  information  now  available  about  the 
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weaknesses  of  paper-and-pencil  tests  for  measuring 
school  and  job  success  had  not  been  published. 
Even  the  most  ardent  supporters  of  checkrooms 
and  PM  in  ATC  had  much  more  faith  in  the  value 
of  such  paper-and-pencil  tests  than  the  subsequent 
research  indicated.  So  under  such  circumstances, 
one  cannot  be  too  critical  of  the  decision-makers 
who  caused  the  elimination  or  deemphasis  of  PM. 
Perhaps,  if  such  information  had  been  presented  at 
that  time,  ATC  would  have  retained  its 
checkrooms  and  its  PM. 

Early  PM  Efforts  of  the  Advanced 
Systems  Division  (AFHRL) 

With  the  abolishment  of  AFPTRC  in  1958  and 
the  resultant  closing  of  its  Maintenance  Labora- 
tory, the  Air  Force  maintenance  research  responsi- 
bility was  transferred  to  the  Behavioral  Sciences 
Laboratory  (BSL)  at  Wright-Patterson  Air  Force 
Base,  Ohio,  but  with  greatly  reduced  manpower 
and  monetary  support.  Since  none  of  the  research 
personnel  were  transferred  with  the  responsibility, 
and  all  of  the  ongoing  projects  had  been  cancelled, 
the  research  program,  conducted  by  the  Training 
Research  Division  of  BSL,  was  not  a true  continua- 
tion of  work  of  the  Maintenance  Laboratory.  (In 
1968,  the  Training  Research  Division  of  BSL 
became  part  of  the  newly  formed  Air  Force 
Human  Resources  Laboratory  and  eventually  was 
renamed  the  Advanced  Systems  Division  (AS)  of 
AFHRL). 

The  maintenance  R&D  supported  by  BSL  and 
its  successor,  AFHRL/AS,  has  been  characterized 
by  its  emphasis  on  the  maintenance  man’s  inter- 
face with  the  hardware  being  maintained,  as  well 
as  the  improvement  of  his  efficiency  of  perform- 
ance on  the  job.  Before  an  extensive  program  was 
started,  an  in-house  analysis  was  made  concerning 
the  variables  that  contribute  to  the  performance  of 
maintenance  (sec  Foley,  1973,  pp.  14-16). 
Eventually  three  closely  related  R&D  programs 
resulted,  namely,  performance  measurement,  job 
performance  aids  (JPA),  and  job  (task)  oriented 
training  (TOT).  In  each  of  these  programs,  a deter- 
mined effort  was  made  to  make  maximum  use  of 
the  previous  R&D  conducted  by  Army,  Navy,  and 
Air  Force  including  the  AFPTRC  woA.  The  plan- 
ning of  new  work  for  each  program  was  preceded 
by  an  in-depth  review  and  analysis  of  the  R&D 
literature. 

in  regard  to  rhe  literature  reviews  and  analyses 
made  for  PM  (Foley,  1967,  1974),  many  valuable 
PM  efforts  have  been  reported  by  the  Army,  Navy, 
and  Air  Force.  However,  most  of  these  efforts 
have  not  been  systematic  efforts,  having  as  their 
prime  objective  the  improvement  of  the  state-of- 


the-art  of  PM.  Rather,  they  have  been  ad  hoc  PM 
developments  to  support  job  oriented  training 
research  programs.  A notable  exception  was  the 
work  of  the  AFPTRC  Maintenance  Laboratory. 
(Another  more  recent  systematic  Army  effort, 
accomplished  by  the  Human  Resources  Research 
Organization  (HumRRO)  was  not  covered  in  these 
reviews  (Vineberg  et  al.,  1970a,  1970b;  Vineberg 
& Taylor,  1972a,  and  1972b).  As  for  civilian  R&D, 
during  the  initial  PM  literature  review  (Foley, 
1967),  a serious  attempt  was  made  to  identify  and 
include  the  results  of  PM  R&D  from  the  civilian 
vocational  education  establishment.  None  was 
found. 

A substantial  outcome  of  the  review  of  other 
PM  efforts  was  a consolidation  of  research  results 
concerning  the  correlations  between  results  of  PM 
for  various  maintenance  tasks  a/irf  paper-and-pencil 
theory  tests,  job  knowledge  tests,  and  school 
marks.  As  to  their  value  for  measuring  ability  to 
perform  maintenance  tasks,  this  research  evidence 
gives  a low  rating  to  all  of  these  paper-and-prencil 
based  measures  of  school  and  job  success.  Table  1 
shows  correlations  that  have  been  obtained  by 
comparing  job  task  performance  tests  (JTPT)  to 
theory  tests  and  job-knowledge  tests.  The  latter 
two  are  paper-and-pencil  tests.  Table  1 also  in- 
cludes correlations  of  JTF*T  with  school  marks.  As 
indicated  earlier,  school  marks  have  been  heavily 
weighted  with  the  paper-and-pencil  test  scores.  An 
examination  of  this  table  indicates  that  the  cor- 
relations of  JTPT  scores  with  theory  test  scores  are 
generally  somewhat  lower  than  with  job- 
knowledge  tests.  None  of  these  measures  are 
sufficiently  valid  for  use  as  substitutes  for  JTPT 
(Foley,  1967, 1974). 


II.  THE  MAN-MACHINE  INTERFACE 
FOR  MAINTENANCE 

As  stated  previously,  the  maintenance  R&D 
supported  by  AFHRL  has  emphasized  the  man- 
machine  interface.  From  this  point  of  view,  PM  for 
all  personnel  associated  with  machine  systems 
must  determine  the  ability  of  such  personnel  to 
perform  tasks  generated  by  the  man-machine  inter- 
face. Although  there  may  be  some  overlap,  most 
of  the  task  functions  demanded  by  a machine 
system  of  its  operator  personnel  are  different  than 
those  task  functions  demanded  of  its  maintenance 
personnel.  Herein,  lies  most  of  the  unique,  distin- 
guishing characteristics  of  PM  for  maintenance.  As 
a result,  this  section  of  my  paper  will  be  devoted 
to  a discussion  of  the  complexity  of  maintenance 
task  functions. 
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Tahk'  / ('ofTelation&  Between  Job-Task  IVifonnaiKe  Tests  and  Theory 
Tests,  Job  Knowledge  Tests,  and  School  Marks 


Type  of  iob  Task 

Theory 

Job  Knewi- 

School 

Rtttarchtes 

Performance  Test  (JTPT) 

Tests 

e<S«e  Tests 

Marks 

Antler  sun 

Test  Eciuipment  JTPT 

.18-  .33 

(1962) 

Evans  and 

Troubleshooting  JTPT 

.24  & 36 

12  & 10 

.36 

Smith  (1963) 

Vlackre  el 

Troubletfiootmg  JTPT 

38 

39 

at  11963) 

Saupe  (1965) 

Troubleshooting  JTPT 

55 

66 

Brown  et  a<  . 

Troubleshooting  JTPT 

40 

11969) 

Test  Equipment  JTPT 

29 

Alignment  JTPT 

28 

Repair  Skills  JTPT 

19 

Milhams  arxJ 

Troubleshootirsg  JTPT 

Whitmore 

(Inexperienced  Subiecis) 

.23 

(1969) 

(Experienced  Subjects) 

16 

Adioltment  JTPT 

(Inexperienced  Subjects) 

.02 

(Experienced  Subiects) 

Acquisition  Rader  JTPT 

.21 

(Inexperienced  Subiects) 

.03 

36 

(Experienced  Subjects) 
Target  Tracking  Radar  JTPT 

14 

22 

(Inexperienced  Subiects) 

24 

33 

(Experienced  Subjects) 
Missile  Tracking  Radar  JTPT 

20 

38 

(Inexperierrced  Subjects) 

09 

15 

(Experienced  Subjects) 
Computer  JTPT 

19 

32 

(Inexperienced  Subjeas) 

06 

24 

(Experienced  Subjects) 

06 

14 

Total  JTPT 

(Inexperienced  Subjects) 

14 

(Experienced  Subjects) 

20 

Crowder  el 

Troub(eshootif>g  JTPT 

If 

18  32 

al  . (1964) 

Past  Human  Factors  Emphasis 

But  before  discussing  the  characteristics  of  task 
functions  for  maintenance,  it  might  be  well  to  call 
attention  to  the  fact  that  human  factors  establish- 
ments have  given  much  more  attention  to  the 
operator  interface  with  machines  than  they  have 
given  to  the  maintenance  personnel  interface. 
Many  actions  are  taken  to  maximize  effective  and 
efficient  performance  of  the  operator.  Work 
stations  are  human-engineered  to  maximize  the 
efficiency  and  comfort  of  the  human  operator. 
Major  training  facilities  are  provided,  so  that, 
operators  can  receive  a large  amount  of  supervised 
practice  in  performing  typical  tasks  of  their  job. 
Graduation  from  training  is  based  primarily  on 
demonstrated  ability  to  perform  job  tasks.  And, 
periodic  checks  are  made  of  the  operator's  ability 
to  perform  the  critical  tasks  of  his  job.  These, 
actions  of  course,  are  not  all  of  the  many  efforts 
made  to  maximiz.c  the  performance  of  human 
operators. 

Generally,  the  human  factors  establishment  has 
given  little  attention  to  the  effectiveness  and 


efficiency  of  the  maintenance  man’s  interface  with 
hardware.  The  maintenance  work,  including  the 
PM  work  of  AFHRL/AS,  has  emphasized  this 
neglected  interface. 

The  Structure  of  the  Man-Machine 
Interface  for  Maintenance 

One  of  the  results  of  our  R&D  for  maintenance 
has  been  the  evolution  and  articulation  of  a 
structure  for  handling  maintence  functions  and 
their  complex  relationships  in  a systematic  man- 
ner. This  structure  includes  (a)  standard  main- 
tenance functions  and  action  verbs,  (b)  a working 
definition  of  a maintenance  task,  and  (c)  schemes 
for  handling  the  complexities  of  maintenance 
tasks. 

Standard  Maintenance  Functions  and  Action 
Verbs.  The  establishment  of  standard  maintenance 
functions  and  action  verbs  has  been  one  of  the 
widely  accepted  results  of  the  Air  Force  Systems 
Command’s  (AFSC)  JPAs  effort  entitled  “Presen- 
tation of  Information  for  Maintenance  and  Opera- 
tion’’ (PIMO).  (Although  the  PIMO  project  was 
managed  by  the  Space  and  Missile  Systems 
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Organi/.alion  (SAMSO)  of  AFSC,  AS  provided 
active  participation  and  technical  inputs  during  the 
entire  project  from  l%6  througli  1969.  AS  has 
incorporated  the  key  findings  and  outputs  of 
PIMO  in  its  own  JPA  efforts.)  Karly  in  the  PlMO 
project,  it  was  found  that  many  maintenance 
action  verbs  and  functions  were  used  by  mainte- 
nance people,  some  with  several  different  mean- 
ings. Part  of  this  confusion  was  caused  by  the 
language  used  in  maintenance  technical  orders 
which  were  written  by  different  people  and  pro- 
duced by  many  different  hardware  manufacturers. 
As  a result,  maintenance  technicians  tliemselves 
did  not  generally  use  precise  language.  A study  was 
made  to  identify  and  define  these  action  verbs. 
Where  two  or  more  verbs  were  used  to  indicate  a 
similar  action,  the  preferred  verb  was  selected, 
based  on  the  expressed  preferences  of  a sample  of 
maintenance  men  with  a wide  range  of  mainte- 
nance AFSCs.  Tlie  use  of  the  preferred  verbs  of 
this  list  is  now  a firm  requirement  of  Air  Force 
technical  order  specifications,  as  well  as  of  recent 
Army  and  Navy  specifications  (see  Joyce  et  al., 
1973,  pp.  97-142). 

A U’ttrking  Definition  of  a Maintenance  Task. 
Within  this  list  of  action  verbs  are  a number  of  key 
action  verbs  (functions).  A key  action  verb,  with 
an  appropriate  specific  hardware  unit  as  its 
predicate,  becomes  a task  statement.  Such  a task 
statement  represents  a maintenance  task  which  can 
be  demanded  by  the  existence  and  operation  of  a 
specific  machine  subsystem.  A list  of  these  func- 
tions is  found  in  AFHRL-TR-7343(1)  (Joyce  et 
al.,  1973,  pp.  19-20).  This  list  includes  functions 
which  arc  found  in  both  mechanical  and  electronic 
jobs.  Some  apply  only  to  mechanical  jobs  and 
some  apply  to  both. 

Schemes  for  the  Systematic  Consideration  of 
Maintenance  Functions  and  Tasks.  Three  schemes 
have  been  developed  for  the  systematic  considera- 
tion of  maintenance  functions  and  tasks  and  the 
key  factors  that  affect  them. 

Scheme  One.  A convenient  model  for  cate- 
gorizing these  maintenance  functions  with  relation 
to  the  type  of  hardware  and  the  level  of  main- 
tenance is  presented  in  Figure  I.  The  common 
maintenance  functions,  already  mentioned  to- 
gether with  the  usage  of  test  equipment  and  hand- 
tools,  arc  represented  on  one  axis  of  the  model. 
Since  mechanical  and  electronic  subsystems 
usually  require  a different  variety  of  maintenance 
actions,  they  arc  represented  by  another  axis.  (In 
regard  to  this  axis,  mechanical  maintenance  could 
be  further  divided  into  two  categories:  (a)  repre- 
sented by  hardware;  such  as  jet  engines,  and  (b)  by 
hardware;  such  as  airframes,  and  tank  and  ship 
hulls). 


The  third  axis  of  the  model  represents  the  three 
levels  or  categories  of  maintenance  now  found  in 
the  military  services.  Organizational  maintenance 
is  the  first  level.  It  is  usually  aimed  at  checking  out 
a whole  machine  subsystem  and  correcting  any 
identified  faults  as  quickly  as  possible.  Flight-line 
maintenance  falls  in  this  category.  A system  is 
checked  out.  If  it  does  not  work,  the  line  replace- 
able unit  (LRU)  or  “black  box”  causing  the  mal- 
function is  identified  and  replaced.  Tliis  major 
component  is  then  taken  to  the  field  shop  (inter- 
mediate maintenance)  where  it  is  again  checked 
out  and  the  faults,  authorized  for  correction,  arc 
corrected.  The  corrective  actions,  authorized  at 
the  intermediate  level,  vary  greatly  from  system  to 
system  depending  on  tlte  maintenance  concept  of 
each  system.  On  some  systems,  the  maintenance 
man  will  troubleshoot  the  black  box  to  the  piece- 
part  level.  In  more  modern  equipment,  he  will 
identify  a replaceable  mixlulc  made  up  of  many 
piece  parts.  Some  modules  arc  thrown  away, 
others  sent  to  the  depot  for  repair.  Any  LRUs 
which  the  field  shop  is  unable,  or  unauthorized,  to 
repair  arc  sent  to  the  depot  for  overhaul. 

Organizational  and  intermediate  level  organiza- 
tions are  manned  primarily  by  enlisted  technicians 
whose  average  length  of  service  is  rather  short 
(sliglitly  more  than  4 years  in  the  Air  Force). 
Depots  arc  manned  largely  by  civilian  personnel 
with  a much  higlicr  level  of  experience  and  longer, 
expected  retention  (service)  time.  Using  this 
model,  it  has  been  possible  to  specify  areas  of 
concentration  for  study. 

Since  PM  requirements  for  maintenance  are  sir 
different  for  the  various  blocks  indicated  in  this 
model,  it  is  extremely  important  that  PM  re- 
searchers indicate  the  precise  blocks  of  their  con- 
centration. To  date,  AFHRL/AS  Iras  concentrated 
on  the  shaded  electronic  portions  of  this  model 
(Figure  I).  Tire  resultant  model  battery  of  48 
JTPT  (together  with  their  symbolic  substitutes) 
will  be  described  later.  In  addition,  a battery  of 
eleven  JTPT  were  developed  on  an  ad  hoc  basis 
(Shriver  & Foley,  1975)  for  mechanical  tasks  at 
the  organizational  level  of  maintenance  (see 
shaded  portion  of  Figure  2).  The  HumRROwork, 
mentioned  previously  (Vineberg  el  al.,  1970a, 
1970b;  Vineberg  & Taylor,  1972a,  1972b)  was 
concerned  with  mechanical  hardware  (tank  and 
truck).  Tire  thirteen  tests  developed  concerned  the 
maintenance  functions  which  are  indicated  by  the 
shaded  portions  of  Figure  3. 

Scheme  T\vo.  Maintenance  functio  ■'  have 
limited  meaning  unless  applied  to  specific  hard- 
ware. A task  identification  matrix  (TIM)  is  an  ex- 
tremely  effective  and  necessary  device  for 
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Figure  1.  A functional  representation  of  the  DOD  maintenance  structure 
(shaded  portion  indicates  scope  of  AFHRL  PM  development  for  electronic  maintenance). 


Figure  2.  A functional  representation  of  the  stops  of  AFHRL  PM 
development  for  mechanical  maintenance. 
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Figure  3.  A functional  representation  of  the  scope  of  the  HumRRO  PM 
development  for  mechanical  maintenance  (Vineberg  et  al.,  1970b). 


interfacing  these  maintenance  functions  with  the 
appropriate  hardware  units  and  thus  identifying 
the  maintenance  tasks  that  arc  generated  by  a 
specific  machine  subsystem  (see  Figure  4).  The 
TIM,  when  properly  structured,  will  reflect  the 
maintenance  level  or  levels  of  interest;  that  is, 
organizational,  intermediate  and/or  depot. 
A FHRL-TR- 73-43(1)  (Joyce  et  al,  1973,  pp. 
16-37)  provides  detailed  directions  for  developing 
a TIM. 

Scheme  Three.  A matter  of  serious  concern 
when  developing  and  structuring  PM  for  main- 
tenance tasks  is  the  interaction  among  the  mainte- 
nance tasks  for  identical  hardware.  A four-level 
hierarchy  of  dependencies  can  be  stated.  Figure  5 
gives  a graphic  presentation  of  these  dependencies 
among  maintenance  activities  for  an  electronic 
hardware. 

The  checkout  of  the  AN/APN-147  (Doppler 
Radar),  for  example,  can  be  a task  in  its  own  right. 
But  the  same  checkout  activity  becomes  an 
element  of  other  major  tasks,  such  as  calibrate. 
Calibration  of  the  doppler  radar  includes  the 
operation  of  specific  general  and  special  test  equip- 
ments, the  use  of  specific  handtools,  as  well  as  the 
checkout  activity.  Troubleshooting  of  an  electron- 
ic equipment,  such  as  AN/APN-147,  requires  the 
use  of  general  and  special  test  equipments.  It  may 


require  remove  and  install  activities  and/or  adjust, 
align,  and  calibrate  activities.  Efficient  trouble- 
shooting practice  usually  requires  the  use  of  a 
cognitive  strategy  to  adequately  track  the  depen- 
dent activities  (but  the  cognitive  strategy  in  itself 
is  not  troubleshooting).  Any  troubleshooting  task 
should  begin  and  end  with  an  equipment  check- 
out. Because  of  these  various  and  varying 
dependency  relationships,  such  activities  as 
checkout,  remove,  install,  disassemble,  adjust, 
align,  calibrate,  or  troubleshoot  cannot  legiti- 
mately be  considered  as  discrete  tasks,  even  for 
one  electronic  system. 

Another  confounding  -factor  is  the  false  cor- 
respondence that  the  same  functional  verbs  create 
when  applied  to  different  electronic  hardware.  For 
example,  personnel  with  the  Avionic  inertial  and 
Radar  Navigation  Systems  Specialist,  AFSC 
328X4,  are  maintaining  at  least  50  major  elec- 
tronic subsystems.  Many  vintages  of  hardware 
design  are  represented.  The  checkout  activity  for 
each  is  different  (both  in  content  and  difficulty) 
and  in  some  cases,  very  different.  The  lack  of  cor- 
respondence of  alignment,  calibration,  and 
troubleshooting  tasks  from  one  specifle  equipment 
to  another  is  even  greater.  An  example  of  the  lack 
of  correspondence  from  one  hardware  to  another 
(both  having  the  same  function)  is  the  wide 
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Figure  4.  Example  of  a task  identification  matrix  (TIM).  Cell  entries:  - (dash)  no  maintenance  task 
of  this  type  is  performed  on  this  hardware  item;  0 - task  of  type,  performed  at  organizational 
level;  I — task,  performed  at  intermediate  level;  and  D - task,  performed  at  depot  level. 


Figure  5.  Indicating  the  dependencies  among  maintenance  functions 
for  electronic  hardware  ( functions  italicized). 


difference  in  the  content  and  difficulty  of  trouble- 
shooting tasks  between  two  doppler  radars.  Tlie 
AN/APN-147,  which  is  used  on  the  C-130  and 
C-141,  has  approximately  14,000  shop  replaceable 
units  (SRU)  whereas  the  inertial  doppler  naviga- 
tion equipment  (IDNE)  on  the  C-5  has  only  28. 
This  lack  of  correspondence  of  functions  across 
electronic  hardware  makes  it  difficult  to  generalize 
from  results  of  PM  from  one  electronic  hardware 
to  another.  One  exception  is  in  the  area  of  general 
test  equipment,  which  may  be  used  in  perfonning 
maintenance  tasks  across  many  hardware 
subsystems. 

The  examples  given  are  characteristic  of  many 
of  the  electronic  maintenance  AFSCs.  Similar 
problems  in  complexity  of  maintenance  functions 
and  tasks  arc  found  in  mechanical  hardware,  but 
to  a lesser  degree. 

Maintenance  Functions  and  Tasks  and 
Traditional  Psychological  Variables 

In  this  consideration  of  the  characteristics  of 
maintenance  functions  and  tasks,  the  psycho- 
logical language  normally  used  by  human  factors 
specialists  in  describing  the  activities  of  operator 
personnel  has  not  been  used.  There  are  several 
reasons  for  this  nonusage.  Such  analyses  would  be 
extremely  expensive  to  generate  and  would  be  of 
little  value  to  maintenance  personnel  and  training 
people.  In  most  cases  a task  (generated  by  a main- 
tenance functional  verb  plus  its  specific  hardware 
unit)  is  considerably  different  from  another  task 
(generated  by  the  same  functional  verb  plus  a 


different  hardware  unit).  A separate  human  factors 
analysis  would  have  to  be  made  for  each  of  these 
tasks.  Some  maintenance  specialties  now  include 
over  50  major  electronic  subsystems-most  of 
which  produce  hundreds  of  such  tasks. 

A traditional  human  factors  type  of  task 
analysis  for  such  tasks,  if  properly  utilized,  would 
probably  be  of  great  value  during  the  original 
design  of  a specific  hardware  or  for  the  design  of 
realistic  training  simulators.  But  must  of  main- 
tenance personnel  interface  with  such  subsystems 
long  after  their  design.  Tlic  type  of  task  analysis 
required  for  the  maintenance  man  calls  for  a dif- 
ferent language.  Tire  functions  used  in  this  dis- 
cussion of  PM  are,  therefore,  based  on  a common 
language  that  is  familiar  to  (if  not  always  com- 
pletely understrwd  by)  a wide  range  of  DOD 
personnel  directly  or  indirectly  associated  with 
maintenance. 


III.  DEVELOPMENT  OF  PM  AND 
SYMBOLIC  SUBSTITUTES  FOR  PM 

Starting  in  1969,  the  Advanced  Systems 
Division  of  the  AFHRL  supported  a modest 
program  to  provide  the  Air  Force  with  the  neces- 
sary tools  for  measuring  the  ability  of  maintenance 
personnel  to  perform  the  key  tasks  of  their 
jobs.  The  scope  of  this  work  was  limited  to  the 
maintenance  of  electronic  hardware  at  the  organi- 
zational and  intermediate  levels  (sec  siraded 
prortion  of  Figure  1).  This  program  had  two 
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objectives;  (a)  to  develop  a model  battery  of  job 
task  perfomiance  tests  (JTPT)  together  with 
appropriate  scoring  schemes  for  the  measurement 
of  the  task  performance  ability  of  electronic  main- 
tenance personnel  (an  effort  was  to  be  made  for 
the  development  of  JTPT  which  could  be  easily 
administered),  and  (b)  using  the  JTPT  of  this 
battery  as  criteria,  to  develop  and  try  out  a series 
of  paper-and-pencil  symbolic  substitute  tests  that 
would  hopefully  have  high  empirical  validity. 

Criterion  Referenced  Job  Task  Performance  Tests 

A model  battery  of  48  criterion  referenced 
JTPT  and  a test  administrator's  handbook  were 
developed  for  measuring  ability  to  perfomi  elec- 
tronic maintenance  tasks.  Copies  of  the  actual 
instructions  for  test  subjects  together  with  the  test 
administrator's  handbook  are  available  from  the 
Defense  Documentation  Center  (DDC)  as 
AFHRL-TR-74-57(ll)  Part  II  (Shriveret  al,  1975). 
The  test  administrator's  handbook  was  developed 
with  step-by-step  detaUed  instructions  so  that  an 
individual  with  a minimum  of  electronic 
maintenance  experience  can  administer  the  tests. 

The  battery  includes  separate  tests  for  the 
following  classes  of  job  activities:  (a)  equipment 
checkout,  (b)  alignment/calibration,  (c)  removal/ 
replacement,  (d)  soldering,  (e)  use  of  general  and 
special  test  equipment,  and  (0  troubleshooting. 
The  Doppler  Radar  AN/APN-147  and  its  Com- 
puter AN/ASN-35  were  selected  as  a typical  elec- 
tronic system.  This  system  was  usee  as  the  test-bed 
for  this  model  battery.  Tire  soldering  and  general 
test  equipment  JTPT  are  applicable  to  all  elec- 
tronic technicians.  The  other  tests  of  the  battery 
apply  to  technicians  concerned  with  this  specific 
doppler  radar  system.  A detailed  description  of  the 
development  and  tryout  of  these  JTPT  is  given  in 
AFHRL-TR-74-57(lI)  Part  1 (Shriver  & Foley, 


1974a).  F.ach  class  of  activity  (for  which  JTPT 
were  developed)  contains  its  individual  mix  of 
behaviors,  but  it  is  not  mutually  exclusive.  As  indi- 
cated in  Figure  5 and  Table  1,  a four-level 
hierarchy  of  dependencies  exists  among  them. 

After  considering  product,  process,  and  time  as 
to  their  appropriateness  for  scoring  the  results  for 
each  activity,  it  was  decided  that  a test  subject  has 
not  reached  criterion  until  he  has  produced  a com- 
plete, satisfactory  product.  This  was  a go,  no-go 
criterion. 

Table  2 summarizes  the  number  of  tests,  prob- 
lems and  scorable  products  by  class  developed  for 
the  AN/APN-147  and  AN/ASN-35.  The  simple 
addition  of  numbers  shown  in  Table  2 indicates 
that  there  are  48  tests,  81  problems,  and  133  scor- 
able products.  But,  these  numbers  tell  us  nothing 
in  terms  of  the  content  of  the  tests.  To  say  that 
one  test  subject  accomplished  100  scorable 
products,  while  another  accomplished  90,  tells  us 
nothing  about  the  job  readiness  of  these  indi- 
viduals or  that  one  is  better  than  the  other.  The 
varieties  of  scorable  products  are  so  diverse  that 
any  combination  of  them,  without  regard  to  what 
they  represent,  is  meaningless.  The  only  meaning- 
ful presentation  of  such  infonnation  must  be  in 
terms  of  a profile  designed  to  attach  meaning  to 
such  numbers.  A sample  of  such  a profile  is  shown 
in  Figure  6. 

This  profile  is  not  presented  as  the  final  solu- 
tion to  the  profile  problem  for  JTPT  for  electronic 
maintenance.  It  does  contain  most  of  the 
important  information  regarding  a test  subject’s 
success  on  the  full  range  of  tests.  It  gives  a 
meaningful  picture  of  the  subject’s  job  task 
abilities  as  measured  by  the  test  battery,  indicating 
the  subject’s  strengths  and  weaknesses. 

An  examination  of  the  profile  (Figure  6)  indi- 
cates that  most  of  the  tests  in  this  battery  contain 
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only  one  problem.  For  example,  there  are  two 
checkout  tests,  having  one  problem  each  and  there 
are  eleven  troubleshooting  tests  having  one  prob- 
lem each.  There  are  two  soldering  tests;  one  has 
two  problems  and  the  other  has  three.  The  volt- 
ohmmeter  (VOM)  test  has  20  problems. 

The  subject  receives  no  “credit”  for  a problem 
unless  he  obtains  all  of  the  expected  products.  No 
attempt  is  made  to  combine  these  scores  in  terms 
of  meaningless  numbers. 

The  hierarchy  of  dependencies  discussed  pre- 
viously (Figure  5)  has  implication  for  the  order  in 
which  tests  are  administered,  as  well  as  for  diag- 
nostics. For  example,  since  troubleshooting  in- 
cludes the  use  of  test  equipment  and  other 
activities  in  the  hierarchy,  logic  would  dictate  that 
in  most  training  situations  the  administration  of 
the  tests  for  the  subactivities  would  precede  the 
troubleshoirting  tests  and  that  a test  subject  would 
not  be  permitted  to  take  the  troubleshtxrting  tests 
until  he  had  passed  these  other  subtests.  Under 
some  circumstances,  one  may  wish  to  reverse  the 
process.  A subject  who  successfully  completes 
selected  troubleshooting  or  alignment  tests  can  be 
assumed  to  be  proficient  in  his  use  of  test  equip- 
ment and  checkout  procedures.  These  depen- 
dencies are  displayed  on  (he  left-hand  side  of  the 
profile  (Figure  6). 

Due  to  the  unavailability  of  a sufficient  number 
of  experienced  test  subjects  at  the  time  of  the 
tryout  of  the  JTPT  battery,  the  tryout  was  not  as 
extensive  as  planned.  The  limited  tryout  did  indi- 
cate that  the  tests,  as  developed,  are  admin- 
istratively feasible.  Their  continued  use,  no  doubt, 
would  result  in  further  modifications  and 
improvements. 

Development  of  Symbolic  Substitutes 

There  is  no  doubt  that  a battery  of  JTPT  would 
require  more  training  and  on-ihe^ob  time  of  the 
test  subjects,  more  equipment,  and  specially 
trained  test  administrators.  It  will  be  recalled  that 
these  were  high  among  the  reasons  given  for  drop- 
ping PM  from  the  Air  Force  and  Navy  mainte- 
nance training  programs.  Therefore,  the 
availability  of  empirically  valid  symbolic  substitute 
tests  would  be  highly  desirable.  Even  though 
previous  attempts  to  develop  such  tests  as  the  Tab 
test  (Crowder  et  al.,  1954)  had  failed,  it  was  our 
opinion  that  much  more  work  could  be  done  to 
improve  symbolic  maintenance  tests  as  substitutes 
for  JTPT.  It  was  hypothesi/.ed  that  higher  correla- 
tions possibly  could  be  obtained  by  a different 
approach  to  the  development  of  symbolic  tests.  A 


study  of  the  Tab  tests  (Crowder  et  al.,  1954,  sec 
Table  1)  indicated  that  the  JTPT  used  as  the  cri- 
terion measures  contained  many  distractions  and 
interruptions  to  the  subject’s  troubleshooting 
strategy  (cognitive  process);  such  as,  using  test 
equipment  to  obtain  test  point  information.  In 
addition  to  such  interruptions  in  the  cognitive 
process,  the  subject  can  obtain  faulty  test  point 
information  by  the  improper  use  of  his  test  equip- 
ment. In  the  symbolic  substitute  Tab  tests,  all  of 
these  potential  pitfalls  of  the  actual  task  were 
avoided.  The  subject  was  given  a printed  test  point 
readout.  It  was  hypothesized  that  the  injection  of 
job  equivalent  pitfalls  into  symbolic  substitutes 
possibly  would  increase  their  empirical  validity. 

Based  on  these  hypotheses,  a battery  of  sym- 
bolic tests  was  developed  under  contract  with  the 
Matrix  Research  Company  of  Falls  Church, 
Virginia.  A companion  graphic  symbolic  test  was 
developed  for  each  of  the  job  activities  for  which  a 
criterion  referenced  JTI^  had  previously  been 
developed.  Based  on  two  limited  validations,  all  of 
the  graphic  symbolic  tests,  with  the  exception  of 
the  symbolic  test  for  soldering,  indicated  sufficient 
promise  to  justify  further  consideration  and  refine- 
ment. Table  3 indicates  the  correlations  obtained 
from  these  validations.  Due  to  a shortage  of  avail- 
able subjects,  the  number  of  pairs  of  subjects  was 
extremely  small.  All  of  these  promising  graphic 
symbolic  tests,  therefore,  must  be  given  more 
extensive  validations  using  larger  numbers  of 
experienced  subjects. 

The  validation  of  any  such  symbolic  test  re- 
quires the  administration  of  a companion  JTPT  as 
a validation  criterion.  As  a result,  a validation  is  an 
expensive  process  in  terms  of  equipment  and 
experienced  manpower.  The  troubleshooting 
symbolic  tests  require  the  most  extensive  refine- 
ment. Several  suggestions  are  made  for  improving 
their  empirical  validity.  A complete  description  of 
these  symbolic  test  efforts  can  be  found  in 
AFHRDTR-74-57(lll)  (Shriver  & Foley.  1974b). 
An  attempt,  also,  was  made  to  develop  video 
symbolic  substitute  tests,  but  this  effort  produced 
no  promising  results.  (Shriver  et  al.,  1974). 

Even  if  graphic  symbtrlic  substitutes  of  higji 
empirical  validity  can  be  produced,  the  use  of 
symbolic  substitutes  will  never,  in  my  opinion, 
dispense  with  the  requirement  for  the  liberal 
administration  of  actual  JTPT  to  maintenance 
personnel.  We  can  never  include  all  aspects  of  an 
actual  performance  of  a task  in  a paper  and  pencil 
symbolic  representation  of  that  task,  but  our  work 
indicates  that  we  can  come  much  closer  than  has 
been  done  in  the  past. 
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Figure  6.  A profile  for  displaying  the  results  obtained  by  an  individual  subject  from  a battery  of  Job  Task 
Performance  Tests  concerning  an  electronic  system  - the  AN/APN-147  and  the  AN/ASN-35.  This  represents 
the  profile  of  an  individual  who  has  successfully  completed  most  of  the  battery. 
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Tahk‘  i Indicate  the  Number  of  Pairs  Used  as  Well 
as  the  and  the  C'urrelalions  Obtained  during  Two 
Small  Validations  of  Symbolic  Tests 
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IV.  CONSOLIDATED  DATA  BASE  TO  SUPPORT  PM 

In  keeping  with  its  man-machine  interface 
orientation,  AFHRL/AS  is  demonstrating  the 
technical  feasibility  of  integrating  five  human 
resources  related  technologies  and  applying  them 
during  weapons  system  development.  This  is  being 
accomplished  under  Project  1959,  “Advanced 
System  for  tlie  Human  Resources  Support  of 
Weapon  System  Development.”  The  five  technol- 
ogies are:  (a)  human  resources  in  design  trade  offs, 
(b)  maintenance  manpower  modeling,  (c)  job  per- 
formance aids,  (d)  instructional  system  design,  and 
(e)  system  ownership  costing. 

One  objective  of  this  program  is  to  determine 
the  data  input  requirements  for,  and  prepare  sfieci- 
fications  for,  a consolidated  maintenance  task 
identification  and  analysis  data  base,  which  will 
support  the  integrated  application  of  these  five 
technologies  in  a weapons  system  development 
program.  We  feel  that  such  a consolidated  data 
base  wOl  contain  most,  if  not  all,  of  the  informa- 
tion which  would  be  required  to  develop  good 
JTPT  provided  the  tests  are  developed  in  keeping 
with  the  technology  described  in  this  paper.  If 
such  a data  base  is  demonstrated  to  be  technically 
feasible  and  if  it  is  routinely  made  a requirement 
in  weapons  system  development  contracts,  it  will 
provide  considerable  assistance  in  developing  main- 
tenance performance  tests  for  new  weapons 
systems. 


V.  PROBLEMS  CONCERNING  THE  RESEARCH, 

DEVELOPMENT  AND  IMPLEMENTATION  OF  PM 

As  stated  previously,  PM  for  maintenance  had 
widespread  usage  in  Adt  Force  and  Navy  main- 
tenance training  programs  during  and  after  World 
War  II.  The  dropping  of  such  tests  from  these 
training  programs  reflected  two  interacting  prime 
factors.  Tire  first  prime  factor  is  a/dcf ; tliat  is.  PM 
for  maintenance  are  much  more  expensive  to 
develop  and  to  administer  than  paper-and-pencil 
theory  and  job  knowledge  tests.  However,  the 
second  factor,  the  general  acceptance  of  such  tests 
as  adequate  substitutes  for  PM,  is  not  a fact  but  a 
widely  held  Mie/.  I use  belief  here  with  the 
precise  meaning  of  something  that  is  held  to  be 
true  without  adequate  proof.  Althou^i  we  now 
have  substantial  hard  data  which  disprove  this 
belief  (see  Table  1),  many  people  seem  to  be 
unaware  of  these  data.  Most  of  the  objections  to 
PM  ignore  the  fact  that  paper-and-pencil  tests  are 
not  valid  measures  of  job  ability.  Such  paper- 
and-pencil  tests  are  not  a bargain.  No  matter  how 
cheaply  they  can  be  administered,  their  results  are 
almost  meaningless  in  terms  of  measuring  ability 
to  perform  maintenance  tasks.  This  state  of  affairs 
has  contributed  to  a number  of  other  problems. 

1 . There  is  a well-developed  paper-and-pencil 
test  technology  which  is  based  on  testing  theory 
which  is  appropriate  for  the  academic  variety  of 
education.  This  technology  has  been  institutiona- 
lized and  is  well  entrenched  in  the  DOD  personnel 
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and  training  systems.  All  education  test  and 
measurement  textbooks  and  courses  reflect  this 
technology.  Psychological  measurement  texts 
emphasize  this  technology.  At  least  two  genera- 
tions of  teachers  and  test  and  measurement 
psychologists  have  been  trained  in  the  use  of  this 
technology  and,  as  a result,  many  have  un- 
questioned faith  in  its  application  to  any  personnel 
measurement  problem. 

Most  of  these  people  are  products  of  the 
academic  world.  Few  have  had  any  “hands-on” 
experience  in  performing  maintenance  tasks.  When 
the  appropriateness  of  their  technology  for  the 
measurement  of  maintenance  ability  is  questioned, 
many  members  of  this  papcr-and-pencil  testing 
establishment  become  threatened  and,  therefore, 
defensive. 

2.  In  spite  of  this  extensive  military  history  of 
usage,  there  is  no  PM  establishment  comparable  to 
the  paper-and-pencil  test  establishment.  There  are 
no  college  test  and  measurement  courses  (even  in 
vocational  education  departments)  which  teach 
PM  technology,  and  there  are  no  textbooks  de- 
voted to  the  subject.  The  vocational  educators 
have  emulated  their  academic  brethren  by  using 
their  measurement  texts.  There  has  only  been  a 
limited  amount  of  systematic  R&D  concerning  the 
development  of  a PM  technology.  Most  of  the 
current  PM  technology  for  maintenance  is  found 
in  DOD  technical  reports. 

3.  Just  as  human  factors  resources  have 
favored  the  operator’s  interface  with  hardware 
over  that  of  the  maintenance  man's  interface,  the 
personnel  and  training  resources  have  heavily 
favored  the  operator.  This  has  been  especially  true 
with  regard  to  the  aircraft  pilot.  DOD  still  contains 
elements  of  a caste  system  which  relegated  the 
maintenance  man  to  the  status  of  a “grease 
monkey.”  This  is  a reflection  of  a deep-seated 
culture  bias  in  our  society  against  any  group  who 
gets  their  hands  dirty  while  earning  their  living. 
This  bias  has  been  extremely  strong  in  the  manage- 
ment and  academic  establishments.  The 
importance  of  the  maintenance  man  and  his  prob- 
lems has  been  consistently  downgraded,  perliaps 
not  by  word,  but  certainly  by  the  allocations  of 
resources.  No  matter  how  costly,  the  operator  has 
always  been  provided  the  necessary  hardware  and 
hardware  simulators,  as  well  as  the  necessary  PM, 
to  ensure  his  ability  to  perform  the  tasks  of  his 
job.  Few  such  facilities  have  been  provided  for  the 
maintenance  function-one  result  has  been  an 
effective  but  incfFicient  and  costly  maintenance 
system.  Costly  maintenance  is  directly  translated 
info  excessive  life  cycle  costs  of  ownership  of 
hardware. 


4.  Success  in  aircraft  pilot  training  and  other 
operator  training  has  always  been  based  on  PM; 
that  is,  demonstrated  ability  to  perform  key  job 
tasks.  Consequently,  these  training  programs  have 
been  designed  to  ensure  success  on  PM.  Such 

training  has  been  characterized  by  an  abundance  ’ 

of  supervised  practice  of  job  tasks.  But  for  main- 
tenance personnel,  paper-and-pencil  theory  and 
job  knowledge  tests  have  been  used  as  the 
principal  means  for  determining  both  the  school 
and  job  success.  As  a result,  maintenance  training 
programs,  both  fomial  courses  and  career  develop- 
ment courses  (CDC),  have  come  to  be  structured 
to  ensure  success  on  paper-and-pencil  tests.  This 
has  resulted  in  the  greater  part  of  many  so-called 
maintenance  courses  taking  on  the  verbal  char- 
acteristics of  academic  education.  This  has 
happened  at  the  expense  of  supervised  practice  of 
job  tasks. 

5.  A like  imbalance  of  empliasis  is  reflected  in 
the  more  stringent  PM  certification  required  of  the 
operator.  A pilot  for  example,  is  certified  on  the 
basis  of  his  demonstrated  perfomiance  before  he  is 
permitted  to  fly  a specific  type  of  aircraft,  and  his 
proficiency  is  checked  periodically  as  long  as  he  is 
required  to  fly  that  aircraft.  But  a maintenance 
man  receives  no  such  certification  of  his  ability  to 
perform  the  maintenance  tasks  required  of  him  by 
the  same  aircraft. 

Rather  than  an  equipment-specific  PM  certifica-  j 

tion,  an  “occupational”  certification  based  on  i 

paper-and-pencil  job  knowledge  tests  has  been 
substituted  for  maintenance  personnel.  Many 
maintenance  “occupations"  cover  a large  number 
of  systems  or  subsystems.  An  individual  main- 
tenance man  usually  works  on  one  or  two  such 
systems  or  subsystems.  Tests  for  occupations  have, 
therefore,  been  general  in  nature.  Most  of  the 
personnel  and  training  measures  for  maintenance 
men  in  all  three  services  have  been  of  the  paper- 
and-pencil  job  knowledge  variety.  However,  the 
Army  now  has  a policy  for  including  PM  on 
specific  job  tasks  in  its  maintenance  personnel 
system  (Maier  et  al.,  1976).  This  policy  is  only  in 
an  early  stage  of  implementation. 

Returning  to  the  pilot/maintenance  man  com- 
parison, it  is  true  that  an  improficient  pilot  might 
destroy  a whole  aircrafl.  Thanks  to  good  checkout 
procedures,  it  is  highly  improbable  that  a main- 
tenance man's  actions  would  cause  the  sudden 
destruction  of  a whole  aircraft.  However,  over  a 
period  of  time  an  improficient  maintenance  man 
can  do  the  equivalent,  on  a piece-by-piece  basis,  by 
the  damage  he  can  cause  by  his  lack  of  skill,  and 
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by  his  consumption  of  unnecessary  spare  parts  to 
correct  malfunctions.  Certification  by  PM  would 
certainly  improve  the  efficiency  of  maintenance. 

6.  Gosely  related  to  this  lack  of  meaningful 
certification  for  maintenance,  is  the  lack  of 
accountability.  The  target  of  the  personnel, 
training  and  tech  data  establishments  should  be  to 
ensure  the  maintenance  man’s  ability  to  perform 
the  tasks  of  his  job  efficiently.  But  our  personnel 
measures  do  not  ascertain  hov/  many  hits  and 
misses  we  make— nor  what  is  causing  our  misses. 
As  a result,  no  one  is  being  held  accountable  for 
the  effectiveness  of  their  contributions  in  terms  of 
efficiency  of  job  task  performance.  Many  people 
in  these  establishments  can  see  no  reason  for 
adopting  improved  technolpgies  such  as  TOT  and 
JPA-because  they  have  never  been  held  account- 
able for  hitting  the  job  performance  target.  We, 
therefore,  require  the  use  of  valid  job  task  per- 
formance measures  to  provide  the  bases  for  such 
required  accountability.  But  such  a possibility 
becomes  very  threatening  to  many  people  in  these 
establishments. 

7.  In  spile  of  all  of  the  evidence  supporting 
requirements  of  PM  for  maintenance,  it  has  been 
extremely  difficult  to  obtain  R&D  funding  for 
efforts  to  advance  the  PM  technology.  In  addition, 
difficulty  has  been  experienced  in  finding  and  re- 
taining Air  Force  professionals  with  the  necessary 
capability  and  interest  to  pursue  an  effective  PM 
R&D  program  for  maintenance.  Such  professionals 
are  necessary,  for  either  an  in-house  or  contractor 
program. 

Few  contractors  have  had  extensive  experience 
or  expertise  in  this  area.  Any  contractual  effort,  to 
be  effective,  must  be  very  carefully  planned  and 
closely  monitored.  I would  anticipate  that  much 
of  the  first  year’s  effort  by  a new  contractor  will 
be  expended  in  a learning  experience  for  his 
people  and  will  not  be  too  productive  for  the  PM 
technology.  Unless  continued  follow-on  work  is 
given  such  a contractor,  his  expertise  is  soon  lost. 
During  Fiscal  Years  1%9,  1970,  and  1971  a total 
of  $239K  in  exploratory  development  funds  was 
obtained  by  AFHRL/AS  for  the  development  and 
tryout  of  PM  and  symbolic  substitutes.  The  con- 
tractor personnel  for  this  effort  developed  con- 
siderable expertise  in  working  with  PM  for 
maintenance  but  they  are  no  longer  with  the 
original  contractor.  The  principal  investigator.  Dr. 
Edgar  L.  Shriver,  is  now  president  of  his  own  firm, 
but  his  two  PM  assistants  arc  no  longer  with  him. 
Any  successful  program  in  this  PM  area  must  be  a 
long  range  program  making  use  of  existing  ex- 
pertise and  aimed  at  expanding  such  expertise. 
“Off  again,  on  again”  efforts,  and/or  jumping  to  a 


new  contractor  with  every  start  will  result  in  little 
improvement  in  PM  technology. 

8.  During  JTPT  and  symbolic  test  develop- 
ment efforts,  several  attempts  were  made  to  share 
the  use  of  operational  hardware  on  a noninter- 
ference basis.  These  experiences  have  indicated 
that  no  matter  how  cooperative  tire  personnel  of 
the  operational  unit,  such  time-sharing  efforts  are 
very  expensive  in  terms  of  wasted  man-hours  of 
highly  paid  R&D  professional  personnel.  For 
successful  results,  the  necessary  hardware  must  be 
assigned  to  the  R&D  project. 

9.  One  of  the  persistent  problems  concerning 
the  administration  of  PM  has  been  getting  main- 
tenance supervisors  to  shed  their  supervisory  role 
and  assume  the  role  of  a disinterested  test  adminis- 
trator. Because  of  their  strong  urge  to  show  and 
help  test  subjects,  most  of  these  people  have 
extreme  difficulty  in  keeping  themselves  out  of 
the  actual  tasks  performance. 

10.  Timewise,  it  certainly  would  be  impossible 
to  administer  a PM  to  a maintenance  man  for 
every  possible  task  that  his  hardware  system  might 
produce.  This  world  of  tasks  and  people  must  be 
sampled.  The  model  PM  described  previously 
provides  a sampling  procedure  based  on  major  task 
functions  such  as  checkout,  align,  adjust,  trouble- 
shoot, etc.  But  even  this  sampling  across  possible 
tasks  resulted  in  48  tests  and  133  scorabic 
products.  It  would  be  impractical  to  give  any  one 
test  subject  all  of  these  48  tests  at  any  one  time. 
Systematic  sampling  schemes  must  be  developed 
across  tests. 

The  purposes  for  which  PM  results  are  to  be 
used  should  be  considered  when  developing 
sampling  schemes.  Such  purposes  of  PM  could 
include  ascertaining  (a)  the  job  task  proficiency  of 
an  individual,  (b)  the  job  effectiveness  of  a training 
program,  and  (c)  the  proficiency  of  a maintenance 
unit.  Each  of  these  purposes  would  require  a dif- 
ferent mix  or  mixes  of  tests  and  people  Some 
suggestions  for  such  samplings  can  be  found  in 
AFHRL-TR-74-57(ll)  Part  I (Shriver  & Foley. 
1974a).  But  it  should  be  remembertJ  that  these 
are  suggestions  that  must  still  be  field  tested. 

In  the  case  of  determining  unit  proficiency, 
some  PM  can  be  administered  by  on-line  observa- 
tion of  tasks  that  are  often  repeated  such  as  check- 
out. There  will  always  be  a requirement  for  off- 
line PM  concerning  critical,  but  seldom  performed 
tasks.  Whether  the  PM  is  performed  on-line  or 
off-line,  the  test  administrator  must  use  the  same 
objective  scoring  procedures,  the  criteria  of  success 
being  acceptable  products. 
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1 1.  Hie  potential  cost  of  PM  in  both  training 
and  field  environments  has  certainly  been  in- 
creased by  the  proliferation  of  hardware  sub- 
systems (especially  electronic)  since  the  early 
1%0’s.  Over  this  period  the  state-of-the-art  has 
been  constantly  changing.  This  has  resulted  in  the 
proliferation  of  many  variations  in  tasks  for  any 
one  task  function.  For  example,  the  alignment 
function  produces  considerably  different  tasks 
from  hardware  to  hardware.  Some  long  range 
actions  are  being  taken  to  reduce  the  number  of 
hardware  having  the  same  functional  use.  Because 
of  the  large  numbers  and  types  of  maintenance 
tasks,  a realistic  system  of  priorities  must  be 
established  for  PM  development.  PM  concerning 
the  use  of  general  test  equipment  would  probably 
have  the  most  immediate  and  widespread  effect  on 
the  quality  of  maintenance.  This  development 
should  be  followed  by  PM  for  systems  and  sub- 
systems having  long  life  expectancies  and  large 
numbers  in  the  field. 

12.  Current  military  grading  systems  must  be 
modified  to  properly  reflect  the  results  obtained 
from  PM  and  symbolic  substitutes.  In  my  opinion, 
the  only  adequate  device  for  presenting  such  re- 
sults is  a profile  similar  to  that  shown  in  Figure  6. 
No  attempt  should  be  made  to  convert  the  content 
of  such  a profile  into  a single  numerical  score.  The 
results  of  PM  should  never  be  combined  with 
paper-and-pencil  test  results. 

Institutionalization  of  New  Technologies 

Getting  newly  developed  technologies  such  as 
PM  institutionalized  is  a perennial  problem, 
especially,  when  a technology  requires  funda- 
mental changes  in  long  existing  programs,  proce- 
dures, and  attitudes  of  entrenched  establishments. 
AFHRL/AS  has  been  involved  in  the  implementa- 
tion of  several  well-developed  and  documented 
technologies,  such  as  job  performance  aids  and 
instructional  systems  design  (ISD)  including  pro- 
grammed instruction  and  job  (task)  oriented 
training.  These  experiences  have  indicated  that  it  is 
extremely  difficult  to  maintain  the  integrity  of  a 
technology  during  its  so-called  implementation. 
Operational  organizations  invariably  attempt  to 
implement  a much  “watered  down”  version  of  the 
technology  and  consequently  obtain  greatly 
“watered  down”  results.  In  some  cases  only  cos- 
metic changes  to  existing  programs  are  reported  as 
implementations.  Currently  it  requires  years  of 
persistent  effort  on  the  part  of  the  research 
community  to  get  a technology  properly 
institutionaliz,ed. 


A mechanism  must  be  developed  for  the  timely 
institutionalization  of  each  new  technology  which 
will  ensure  its  integrity.  A mechanism  for  the 
orderly  implementation  of  technologies  similar  to 
that  used  for  new  weapons  systems  is  recom- 
mended. Such  a mechanism  must  make  efficient 
and  effective  use  of  the  “know-how”  of  the 
developers  of  the  technology  and  make  them 
responsible  and  accountable  for  its  implementa- 
tion. A new  technology  should  not  be  turned  over 
to  a using  command  for  its  operation  until  it  is  in 
place,  “debugged”  and  operational— just  as  a new 
weapons  system  is  not  turned  over  to  an  opera- 
tional command  until  it  has  been  “debugged”  and 
proven  to  be  ready  for  operational  use. 


VI.  PROPOSED  PM  RAD  EFFORTS 
FOR  MAINTENANCE 

Excessive  maintenance  costs  are  never  going  to 
be  reduced  as  long  as  we  don’t  have  JTPT  and/or 
empirically  valid  symbolic  substitutes  to  ascertain 
how  efficiently  maintenance  men  perform  the 
tasks  of  their  jobs.  In  my  opinion,  the  lack  of  such 
measures  of  maintenance  performance  is  a most 
serious  deficiency  in  DOD.  As  such,  R&D  in  this 
area  should  have  an  extremely  high  priority. 

Areas  for  R&D  Concentration 

For  a long  range  R&D  effort,  five  general  areas 
of  concentration  are  recommended;  namely,  JTPT 
and  matching  symbolic  substitute  tests  for  elec- 
tronic maintenance,  JTPT  and  matching  symbolic 
substitute  tests  for  mechanical  maintenance,  and 
aptitude  tests  based  on  PM.  The  development  and 
field  tryout  of  a JTPT  must  precede  the  develop- 
ment of  its  symbolic  substitute.  The  work  on 
JTPT  batteries  for  both  electronic  and  mechanical 
maintenance  should  be  started  as  soon  as  possible. 
The  work  on  aptitude  tests  should  not  be  started 
until  JTPT  batteries  and  the  symbolic  substitute 
tests  have  been  completely  field  tested.  More  in- 
formation concerning  these  areas  of  concentration 
follows: 

1 . Refinement  of  Model  JTPT  Battery  (Elec- 
tronic Maintenance).  The  already  available  model 
JTPT  Battery  (Shriver  et  al.,  1975)  should  be  given 
a large  scale  field  tryout.  (The  AB328X4  Avionics 
Inertial  and  Radar  Navigation  Systems  Specialist 
Course,  which  includes  the  AN/APN-147  and  the 
AN/ASN-35,  does  not  emphasize  the  mastery  of 
job  tasks.  The  tasks  specific  tests  of  this  battery 
cannot  be  used  in  the  formal  course.)  One  thrust 
of  this  effort  should  be  to  further  refine  the 
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batteiy  including  its  administrative  procedures.  A 
second  thrust  should  be  the  development  of 
samplirig  strategies  which  would  be  appropriate  for 
determining  the  effectiveness  of  training  programs 
and  both  individual  and  unit  proficiency  as  dis- 
cussed earlier  under  PM  problems.  This  effort 
would  require  approximately  two  professional 
man-years  plus  the  use  of  maintenance  specialists 
as  test  administrators  from  the  appropriate  main- 
tenance specialties.  If  it  is  necessary  to  select  a 
system  other  than  the  AN/APN-147-AN/ASN-3.S 
combination,  this  work  would  require 
approximately  four  professional  man-years. 

2.  Refinement  of  Symbolic  Substitutes  (Elec- 
tronic Maintenance ).  As  previously  indicated,  a 
number  of  symbolic  substitutes  for  JTPT  were 
developed  and  given  a limited  tryout.  Table  3 indi- 
cated that  some  of  the  symbolic  tests  show 
promising  empirical  validity.  These  promising 
symbolic  tests  must  be  more  thoroughly  refined 
and  validated.  In  addition,  further  exploratory 
development  is  required  for  symbolic  substitute 
tests  for  troubleshooting  tasks  in  keeping  with 
recommendations  made  in  AFHRL-TR-74-57(IIl) 
(Shriver  & Foley,  1974b).  This  effort  would 
require  between  three  and  four  professional  man- 
years  p/us  the  use  of  maintenance  specialists  as  test 
administrators  and  test  subjects  from  the 
appropriate  maintenance  specialties. 

3.  Development  of  Mode!  JTPT  Battery 
(Mechanical  Maintenance ).  A model  JTPT  battery 
Similar  to  the  model  battery  for  electronic  main- 
tenance described  previously  should  be  developed 
for  a typical  mechanical  subsystem  such  as  a jet 
engine  or  tank  engine  covering  both  the  organiza- 
tional and  intermediate  levels  of  maintenance.  This 
model  should  be  thoroughly  field  tested.  Sampling 
strategies  as  indicated  for  the  electronic  battery 
should  also  be  developed.  This  effort  will  require 
approximately  four  professional  man-years  plus 
the  use  of  maintenance  men  from  the  appropriate 
maintenance  specialties  as  test  administrators  and 
test  subjects. 

4.  Development  of  Symbolic  Substitutes 
(Mechanical  Maintenance ).  An  attempt  should  be 
made  to  develop  symbolic  substitute  tests  with 
high  empirical  validity  after  the  model  JTPT 
battery  is  available  for  mechanical  maintenance. 
The  same  contractor  should  develop  these  sym- 
bolics that  developed  the  JTPT  batteiy.  A very 


rough  estimate  for  accomplishing  this  symbolic 
effort  would  be  four  professional  man-years. 

5.  Job  Aptitude  Test  Research  Based  on 
Results  on  JTPT.  R&D  plans  should  be  made  to 
utilize  the  results  of  JTPT  and  symbolic  substitute 
tests  for  standardizing  military  aptitude  indices 
obtained  from  the  Armed  Service  Vocational  Apti- 
tude Battery  (ASVAB).  As  a first  step,  the  military 
aptitude  scores  of  all  tests  subjects  used  for  the 
tryouts  in  the  proposed  JTPT  R&D  should  be 
recorded.  In  addition,  such  aptitude  scores  should 
be  obtained  during  any  school  or  field  administra- 
tion of  JTPT  or  symbolic  substitutes.  When  suf- 
ficient data  are  obtained,  the  degree  of  relation- 
ship between  JTPT  results  and  various  aptitude 
indices  should  be  obtained.  Later,  when  a 
sufficient  number  of  JTPT  are  used  in  the  field,  a 
formal  R&D  project  should  be  initiated  to  modify 
the  ASVAB  to  directly  reflect  job  success  as 
measured  by  JTPT. 

R&D  Strategy.  Probably  the  most  cost-effective 
approach  for  PM  for  both  electronic  and 
mechanical  maintenance  would  be  to  concentrate 
on  the  development  and  refinement  of  JTPT  on 
use  of  key  test  equipments  prior  to  proceeding 
with  the  other  task  functions  of  the  proposed 
model  test  batteries.  As  indicated  in  Figure  5,  the 
use  of  general  test  equipment  is  a prerequisite  to 
maintenance  task  functions  such  as  alignment, 
calibration  ' :*  troubleshooting.  In  addition, 
general  test  equipments  usually  have  wide  usage  in 
such  task  functions  across  many  hardware  systems 
and  there  are  substantial  amounts  of  data  which 
indicate  that  many  maintenance  men  are  weak  in 
their  test  equipment  ability.  So,  a general  improve- 
ment in  ability  to  use  test  equipment  is  an 
important  and  necessary  factor  for  the  general 
improvement  of  several  maintenance  task 
functions.  I would  strongly  recommend,  therefore, 
that  the  early  concentration  for  the  proposed 
model  test  batteries  be  in  JTPT  concerning  the  use 
of  key  test  equipments.  Each  PM  development  for 
a test  equipment  should  be  accompanied  by  the 
development  of  a pio^ranimed  training  package 
with  sufficient  practice  frames  for  teaching  the 
mastery  of  all  its  functions.  Basic  models  of  such 
training  packages  for  12  general  test  equipments 
are  now  available  (see  Scott  & Joyce,  1975a 
through  1975  1).  However,  more  practice  frames 
should  be  included  in  these  programs. 
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